Add WatsonX metrics test #985
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bdattoma
requested review from
jgarciao,
FedeAlonso,
lugi0,
manosnoam,
nehachopra27 and
tarukumar
FedeAlonso
requested changes
Oct 24, 2023
ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot
Outdated
Show resolved
Hide resolved
ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot
Outdated
Show resolved
Hide resolved
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
bdattoma
force-pushed
the
f/watsonx-metrics
branch
from
3f68c94 to
2dee493
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Robocop found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.
Signed-off-by: bdattoma <[email protected]>
bdattoma
added
needs testing
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
Signed-off-by: bdattoma <[email protected]>
|
verified at |
bdattoma
added
verified
|
Kudos, SonarCloud Quality Gate passed!
|
|
@bdattoma why don't you enable metric in Suite setup. I don't see any harm in that those are cluster wide configuration and having it enable by default will be good idea. |
I thought to have scenarios running without metrics enabled, since users won't be forced to enable them. But yeah I can add it to suite setup, do you mind if we move it there in another PR? this one has been already tested |
tarukumar
approved these changes
Oct 26, 2023
jgarciao
pushed a commit
to jgarciao/ods-ci
that referenced
this pull request
Nov 6, 2023
* add initial test for model metrics Signed-off-by: bdattoma <[email protected]> * add checks for query type and cpu usage Signed-off-by: bdattoma <[email protected]> * remove commented code Signed-off-by: bdattoma <[email protected]> * add kw to protected list Signed-off-by: bdattoma <[email protected]> * add docs and minor fixes Signed-off-by: bdattoma <[email protected]> * add polarion id Signed-off-by: bdattoma <[email protected]> * fix some alerts Signed-off-by: bdattoma <[email protected]> * increase metric check stability Signed-off-by: bdattoma <[email protected]> * fix expected model response Signed-off-by: bdattoma <[email protected]> --------- Signed-off-by: bdattoma <[email protected]>
ChughShilpa
pushed a commit
to ChughShilpa/ods-ci
that referenced
this pull request
Nov 28, 2023
* add initial test for model metrics Signed-off-by: bdattoma <[email protected]> * add checks for query type and cpu usage Signed-off-by: bdattoma <[email protected]> * remove commented code Signed-off-by: bdattoma <[email protected]> * add kw to protected list Signed-off-by: bdattoma <[email protected]> * add docs and minor fixes Signed-off-by: bdattoma <[email protected]> * add polarion id Signed-off-by: bdattoma <[email protected]> * fix some alerts Signed-off-by: bdattoma <[email protected]> * increase metric check stability Signed-off-by: bdattoma <[email protected]> * fix expected model response Signed-off-by: bdattoma <[email protected]> --------- Signed-off-by: bdattoma <[email protected]>
ChughShilpa
pushed a commit
to ChughShilpa/ods-ci
that referenced
this pull request
Jan 2, 2024
* add initial test for model metrics Signed-off-by: bdattoma <[email protected]> * add checks for query type and cpu usage Signed-off-by: bdattoma <[email protected]> * remove commented code Signed-off-by: bdattoma <[email protected]> * add kw to protected list Signed-off-by: bdattoma <[email protected]> * add docs and minor fixes Signed-off-by: bdattoma <[email protected]> * add polarion id Signed-off-by: bdattoma <[email protected]> * fix some alerts Signed-off-by: bdattoma <[email protected]> * increase metric check stability Signed-off-by: bdattoma <[email protected]> * fix expected model response Signed-off-by: bdattoma <[email protected]> --------- Signed-off-by: bdattoma <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is adding a test for checking model metrics for WatsonX/Kserve.
The PR checks:
tgi_,caikit_andistio_are present in the Openshift metrics# of inference requests over defined time period,# of successful inference requests over defined time periodandCPU UtilizationThis is a base for the final test which should check that all the expected metrics are present (waiting for a final list) and will check also more derived metrics (_waiting for a final list and their formulas)